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Abstract 

Random Threshold Networks (RTNs) are an idealized model of di- 
luted, non-symmetric spin glasses, neural networks or gene regulatory 
networks. RTNs also serve as an interesting general example of any 
coordinated causal system. Here we study the conditions for maximal 
information transfer and behavior diversity in RTNs. These conditions 
are likely to play a major role in physical and biological systems, per- 
haps serving as important selective traits in biological systems. We 
show that the pairwise mutual information is maximized in dynami- 
cally critical networks. Also, we show that the correlated behavior di- 
versity is maximized for slightly chaotic networks, close to the critical 
region. Importantly, critical networks maximize coordinated, diverse 
dynamical behavior across the network and across time: the informa- 
tion transmission between source and receiver nodes and the diversity 
of dynamical behaviors, when measured with a time delay between the 
source and receiver, are maximized for critical networks. 
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1 Introduction 



The random Boolean networks (RBNs) model was initially introduced as 
an idealized model of genetic regulatory networks [1]. Since then, the RBN 
model has attracted much interest in a wide variety of fields, ranging from 
cell differentiation and evolution to social and physical spin systems. The dy- 
namics of RBNs can be classified as ordered, chaotic, or critical, as a function 
of the average in-degree k and the bias p in the choice of Boolean functions 
[2] . The central issue of the research on the RBN model is the characteriza- 
tion of the critical transition between ordered and chaotic phases [3] . These 
two regimes produce very different emergent dynamical behaviors. Net- 
works operating in the ordered regime are intrinsically robust, but exhibit 
simple dynamics. This robustness is reflected in the dynamical stability of 
the network both under structural perturbations and transient perturba- 
tions. In contrast, networks in the chaotic regime are extremely sensitive to 
small perturbations, which rapidly propagate throughout the entire system. 
The phase transition between the ordered and chaotic regimes represents a 
tradeoff between stability and access to a wide range of dynamic behavior 
to respond to a variable environment. 

Recently it has been shown that the pairwise mutual information ex- 
hibits a jump discontinuity at the critical values of k and p [4]. Here, we 
extend these results to a second class of discrete dynamical networks called 
Random Threshold Networks (RTN), which were first studied as diluted, 
non-symmetric spin glasses, neural networks and gene regulatory networks 
[5-9]. More specifically, we study the conditions for maximal information 
transfer and behavior diversity in RTNs. We show that the pairwise mutual 
information is maximized in critical networks. Also, we show that the corre- 
lated behavior diversity is maximized for slightly chaotic networks, close to 
the critical region when measured with no delay between source and receiver 
nodes. In contrast, when the delay between the measurement of source and 
receiver nodes is increased, correlated behavior complexity and diversity is 
maximized for critical networks. These results support the hypotheses that 
critical networks provide an optimal information transfer between the ele- 
ments of the network, and optimal coordination of diverse behavior when 
there is a time delay between source and receiver nodes, while slightly chaotic 
networks have an optimal capacity for coordinating most diverse dynamical 
behavior in the absence of such a delay. 
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2 RTN model 



The RTN model consists of randomly interconnected binary variables 
(spins) with states ai G {±1}, i = 0, ...,N — 1 [9]. Each variable has associ- 
ated a function: 

N-l 

'Pi{t)=Y.Uijaj{t) + 9, (1) 
j=o 

where the interaction weights take discrete values uJij € {il} with equal 
probability, and the discrete threshold 9 is fixed. Without loosing generality, 
in the following discussion the threshold parameter is set to ^ = 0. If node i 
does not receive signals from node j, one has Uij = 0. The average number 

k of non-zero interaction weights represents the average connectivity (in- 
degree) of the nodes in the network. The variable cxj will change its state at 
each time step according to the rule: 

Ui{t + l) = sign{^i{t)). (2) 

where sign{x) = 1 if x > and sign{x) = — 1 if x < 0. 

3 Information transfer and behavior diversity 

The average pairwise correlation has been used to characterize the typical 
dynamics of pairs of nodes [9]. The average correlation between two nodes 
i and j is defined as: 

(3) 

where T is the length of the time series over which the correlation is mea- 
sured. If the dynamical activity of two nodes i and j is (anti-)correlated, i.e. 
if (Tj(t) and (Jj{t) always have either the same or the opposite sign, one has 
Cij = 1. If the relationship between the signs of (Ti{t) and crj{t) occasionally 
changes then < Cij < 1. It has been shown that the quantity: 

{C) = N-^Cij), (4) 

(averaged over the RTN ensemble) exhibits a second order phase transition 
at a critical average connectivity kc = 2 [9]. For k < kc, the nodes are 
typically frozen, with {Cij) ~ 1. For k > kc, in the limit of large A^, (Cjj) 
undergoes a transition at kc, vanishing for larger k » kc [9]. 



J2c7iit)ajit) 



t=0 
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Now, let us define the average activity Ai of a node i as following: 



A 



I 

t=o 



(5) 



Frozen nodes, which do not change their states have an activity Aj = 0, while 
the nodes who occasionally change their state have an activity < < 1. 
The average activity 

{A) = iV-i (A) (6) 

is largest (A) ~ 1 in the chaotic phase k > kc, and by decreasing k undergoes 
a second order phase transition at k^ vanishing for k = 0. 

We can define the entropy [10] of the node i, i = 0, 1, N — 1, as: 

Hi = - E Pa{<^i{t))iog2Pai<Ji{t)), (7) 
ae{±l} 

where Paicnit)) is the probability of the symbol a £ {±1} in the time series 
{ai{t)}, i = 0,1, ...,N — 1. The average entropy of the RTN ensemble is: 

{H) = N-'(Y.H^y (8) 

Both, the activity A and the entropy H measure the diversity of the dy- 
namics of the nodes in the RTN ensemble as a function of their connectivity 
k. 

The mutual information [11] between the nodes i and j as a function of 
the time lag r = 0, 1, 2, ... is defined as following: 

a€{il}/36{^l} V Pa{<^i{t))Pp[^At)) J 

(9) 

Here, Pa{o'i{t)) and P/3{<7i{t + r)) are the probabilities of the symbols a, /3 G 
{±1} in {(Tiit)} and respectively {(Tj{t)}, and Papi'^ii't), crj{t + t)) is the 
probability of the pair (a, (3) in {{ai{t),aj{t + r))}. hjiT) measures the 
extent to which information about node i at time t influences node j at 
time t + T. The propagation of information may be indirect, i.e. both nodes 
are influenced by a common node through previous time steps. The above 
probabilities can be easily estimated as following: 

Y.l=o5{a,{t);a)5{a,{t + T);(5) 



Pai3{criit),aj{t + T)) 



Eae{±i} E^e{±i} I^t=o'^i^ii't);a)S{(^j{t + t); /3) ' 

(10) 
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Pa{o-i{t))= Pap(.0-i{t),aj{t + T)), (11) 

/3e{±i} 

ppiai{t))= Pap{cTiit),aj{t + T)), (12) 

ae{±l} 

where 

is the Dirac delta function, and T is the length of the considered time se- 
ries. In order to characterize the information propagation through the entire 
network, wc define the average pairwise mutual information for the RTN en- 
semble as following: 

{I{T))=N-'/Y.I^^i^))■ (14) 

Because the number of pairs that contribute significantly to the sum is 
expected to be at most of order N, it is convenient to work with the quantity 

In{t) = N {I{t)) , (15) 

which approaches a nonzero constant in the large- limit [4]. 

One can define the pairwise correlated behavior in several ways: 

1. as the product between the activity of the nodes and their correlation: 

Dij = liAi + Aj)Cij, (16) 
with the ensemble average given by: 

{D) = iV-2 (Ai) . (17) 

2. as a product between the entropy of the nodes and their correlation: 

^ij = 2iHi + Hj)Cij, (18) 
with the ensemble average given by: 

(F) = iV-2 {Fi,) . (19) 
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3. as the product between the entropy of the nodes and their mutual 
information: 

Qijir) = \{Hi + Hj)Iij{T), (20) 
with the ensemble average given by: 

Qiv(r) = \n-^ m + Hj)Iij{T)) . (21) 

Obviously, these quantities measure the correlated behavior diversity. 
For example, Dij and Fij are high only if both the activity (entropy) of the 
nodes and their correlation are simultaneously high. This means both that 
the nodes are changing, and that they are changing in a correlated fashion. 
Also, assuming that mutual information is a correlation measure, we may say 
that Qij (r) measures the diversity of correlated dynamical behavior between 
the present t and the future at t + r. Thus, by locating the ensemble of 
networks with a connectivity k which maximizes these measures, we have 
found the dynamical region which exhibits the most correlated behavior 
diversity in the present (for D, F), and between the present and the future 
(for Q(t)). 



4 Numerical results 

Since the above defined quantities are not yet analytically accessible, numer- 
ical simulations are necessary for their estimation. The simulation procedure 
is similar to the one described in [3, 4]. For networks of size N = 10^, we 
simulated the dynamics for 10^ steps to eliminate the transient dynamics, 
and collected a time series of length T = 10^. Also, we averaged over 10^ dis- 
tinct, randomly generated networks with 100 runs from different randomly 
chosen initial states for each network. We should note that the ensemble 
averages are computed using a Monte Carlo procedure which includes data 
from all the attractors generated in the calculation. For example, in order to 
calculate (C) we sample Cij randomly over S = 10^ networks and R = 100 
runs from different initial states, and we discard the transient dynamics for 
each run. The sampling is done by computing Cij for M = lOOiV randomly 
generated pairs m = random{i,j), in each run. All the obtained samples 
Cm are then used to calculate an approximation of the ensemble average as: 

S R M 
s=l r=l m=l 
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In Figure 1 we give the average activity (A) , the average correlation (C) , 
the entropy (H) and their numerical derivatives with respect to k: d {A) /dk, 
\d{C) /dk\, d{H) /dk. One can see that all three quantities exhibit a phase 
transition around k'^ ~ 2.2. The deviation from the large-network-size limit 
value of fee = 2, when A/" — ^ oo, is due to the finite size of the simulated 
networks: Akc = k'^. — kc ^ 0.2. 

The average correlated behavior defined by {D) and (F) is shown in 
Figure 2. Both {D) and {F) reach their maximum value in the slightly 
chaotic regime, around k'j^p ~ 2.6 {koF — 2.4, corrected for the finite size 
effect, as noted above). 

In Figure 3 we show the average mutual information In{t) ^ function 
of the time lag r = 0, 1, 40. The mutual information also has a maximum 
value around the critical value k'^ ~ 2.2 {kc — 2.0, corrected for the finite 
size effect). It is interesting to note that InIt) decreases with r, having 
a maximum value for r = 0. This shows that information gained about 
the state of one node given the state of another node decays over time, 
as one might expect. In addition, the numerical simulation suggests that 
/iv(''") is localizing around the critical value kc when the time lag r increases, 
converging to a delta function for large r: /7v(t) — > 6{k, kc)- 

The average correlated behavior Qn{t) is given in Figure 4. The maxi- 
mum of (5 iv(T) shifts to the left when r increases, from k'^ ~ 2.4 when r = 0, 
io kc — 2.2 when r = 40, and it seems to converge to kc for large r. 

5 Discussion and conclusion 

Shannon information measures the information transmission down a noisy 
channel with a decoder of indefinite computational power and seeks to maxi- 
mize information transmission [10] . Cells, other biological and other physical 
systems, do not have decoders of arbitrary power. More, it seems plausible 
that in cells, neural systems, and other tissues, natural selection will have 
acted to maximize both information transfer across the network, and the 
diversity of complex behaviors that can be coordinated within a causal net- 
work. We have shown that the pairwise mutual information is maximized in 
critical RTNs. Also, we have shown that the diversity of complex correlated 
behavior is maximized for slightly chaotic RTNs, close to the critical re- 
gion using two measures of correlated diversity, D and F, with no temporal 
delay between signaling and receiving nodes. Importantly, in the presence 
of a delay, r, between signaling and receiving nodes, maximum diversity of 
complex coordinated behaviors clearly shifts towards critical networks as the 
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delay increases. Ordered networks have convergent trajectories, and hence 
"forget" their past; chaotic networks show sensitivity to initial conditions, 
and thus they, too, forget their past and are unable to act reliably. Critical 
networks, with trajectories that, on average, neither diverge or converge, 
seem best able to bind past to future. In short, our results show that in the 
presence of a delay, hence time binding, critical networks maximize informa- 
tion transfer between source and receiver nodes, i.e. they maximize pairwise 
mutual information, and simulataneously maximize the diversity and com- 
plexity of behaviors that can be correlated by that information transfer. 

Given the potential biological implications, it is of interest that recent 
data suggest that genetic regulatory networks in eukaryotic cells are dynam- 
ically critical [12-14]. Also, recent experiments conducted on rat brain slices 
show that these neural tissues are critical [15]. RTN are simple Boolean 
models of threshold neural networks. Further work with random Boolean 
networks, RBN, will attempt to extend these results to this class of disor- 
dered causal systems, and will extend these results to communication be- 
tween networks. We note that recent results have shown that critical RBN 
maximize power efficiency [3, 16]. Maximum energy efficiency occurs if work 
cycles are performed infinitely slowly. Cells must do work cycles to repro- 
duce. Infinitely slow cell reproduction would fail in the Darwinian race. 
Maximum power efficiency occurs at a finite, defined, displacement from 
equilibrium. Our hope is that subsequent work will establish that cells and 
tissues, as evolved, far from equilibrium evolved systems, simultaneously 
maximize information transfer, the complexity and diversity of dynamical 
behaviors that can be coordinated, and the power efficiency with which these 
complex diverse behaviors arc carried out. Such results may help formulate 
a far from equilibrium theory for living systems. 

Using random threshold boolean nets as simple models of complex causal 
systems we have studied information transfer between source and receiver 
nodes. We have shown that critical RTN maximize both information trans- 
fer, and the complexity and diversity of dynamical behaviors that can be 
coordinated across the network and time. 
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Figure 1: The average activity (A), the average correlation (C), the entropy 
(H) and their numerical derivatives with respect to k: d (A) /dk, d (C) /dk, 
d (H) /dk. 



11 




ure 2: The average activity {A), the average correlation (C), the entropy 
and the average correlated behavior defined by (D) and (F). 
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Figure 3: The average mutual information In{t) as a function of the time 
lag r = 0,1,.. .,7. 
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Figure 4: The average correlated behavior QNiT) as a function of the time 
lag r = 0,1,.. .,7. 
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